One more example:

$ ARRAY_API_TESTS_MODULE=array_api_compat.torch pytest array_api_tests/test_array_object.py::test_getitem

.... snip ....


array_api_tests/test_array_object.py:78: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

shape = (0, 2), dtype = torch.bool, data = data(...)

    @given(shape=hh.shapes(), dtype=hh.all_dtypes, data=st.data())
    def test_getitem(shape, dtype, data):
        zero_sided = any(side == 0 for side in shape)
        if zero_sided:
            x = xp.zeros(shape, dtype=dtype)
        else:
            obj = data.draw(scalar_objects(dtype, shape), label="obj")
            x = xp.asarray(obj, dtype=dtype)
        note(f"{x=}")
        key = data.draw(xps.indices(shape=shape, allow_newaxis=True), label="key")
    
        repro_snippet = ph.format_snippet(f"{x!r}[{key!r}]")
     ###   breakpoint()
    
        try:
>           out = x[key]
E           ValueError: step must be greater than zero
E           
E           ========== FAILING CODE SNIPPET:
E           tensor([], size=(0, 2), dtype=torch.bool)[(slice(None, None, None), slice(1, 0, -1))]
E           ====================
E           
E           Falsifying example: test_getitem(
E               shape=(0, 2),
E               dtype=torch.bool,  # or any other generated value
E               data=data(...),
E           )
E           x=tensor([], size=(0, 2), dtype=torch.bool)
E           Draw 1 (key): (slice(None, None, None), slice(1, 0, -1))
E           Explanation:
E               These lines were always and only run by failing examples:
E                   /home/br/repos/array-api-tests/array_api_tests/test_array_object.py:109

array_api_tests/test_array_object.py:92: ValueError

• test_array_object.py
• test_constants.py
• test_creation_functions.py
• test_data_type_functions.py
• test_fft.py
• test_has_names.py
• test_indexing_functions.py
• test_inspection_functions.py
• test_linalg.py
• test_manipulation_functions.py
• test_operators_and_elementwise_functions.py
• test_searching_functions.py
• test_set_functions.py
• test_signatures.py
• test_sorting_functions.py
• test_special_cases.py
• test_statistical_functions.py
• test_utility_functions.py

The shape of this POC is getting clear now. I've converted one of the test modules, test_manipulation_functions.py. The patch size is mostly whitespace/indentation, and is best viewed with the "hide whitespace" option.

Converting the rest is a bit of mechanical work, but is totally doable.

Downsides:

• git blame gets a bit messier, will need git blame -M -C -C or some such to help git see past trivial indentation changes.
• this will only catch failures in the functions under tests and test assertions. Tricky failures inside hypothesis (either "essential functions" of Providing reproduction code for failed test cases #379 (comment) or obscure "flaky" failures of Skipped flaky tests #300) will not be caught.

I found myself wishing for this feature again today. The added DevEx far outweighs the obscured git blame, in my opinion. Is there anything blocking this?

Thanks for the ping @cbourjau !

• Which function(s) did you hit this with?
• Is the current use of repr appropriate for you? (it makes a difference in numpy, to tell numpy scalars apart from python scalars--- is the f"{arg!r}" good enough for ndonnx?)

I suppose the only thing blocking this is some elbow grease, would you be able to help? Would be great to split the set of test files, #384 (comment) --- the changes are pretty mechanical, just need to be test by test source file by source file.

The solution to the git blame issue is to add a .git-blame-ignore-revs file to the repo root:
https://docs.github.com/en/repositories/working-with-files/using-files/viewing-and-understanding-files#ignore-commits-in-the-blame-view, here's the SciPy usage: https://github.com/scipy/scipy/blob/main/.git-blame-ignore-revs

Therefore,

• it's best to work source file by source file
• to keep the number of entries under control, we should either use a commit per source file or a PR per source file + squash-merge.
• we add the commit hashes to .git-blame-ignore-revs

Coincidentally, I was looking at the test_repeat output today. __repr__ is fine for us. Doing one PR per file sounds better to me since it provides incremental value even if some files have not been migrated yet. I am happy to chime in.